Papers with long-context language models
Literary Evidence Retrieval via Long-Context Language Models (2025.acl-short)
Copied to clipboard
| Challenge: | a recent study shows that long-context language models can exceed human expert performance in literary analysis . despite their speed and apparent accuracy, even the strongest models struggle with nuanced literary signals and overgeneration. |
| Approach: | They propose a task where a model is given an entire text of a book and a literary criticism with a missing quotation from that work and asked to generate the missing quote. |
| Outcome: | The proposed model outperforms open-weight models in literary evidence retrieval tasks. |
DocFinQA: A Long-Context Financial Reasoning Dataset (2024.acl-short)
Copied to clipboard
| Challenge: | Existing work on automating financial numerical reasoning focuses on unrealistically specific document snippets, failing to reflect the broader and more realistic scenarios faced by analysts. |
| Approach: | They propose a long-document financial QA task that augments 7,437 questions from existing FinQA dataset with full-document context, extending the average context length from under 700 words in FinQA to 123k words in DocFinQA. |
| Outcome: | The proposed task extends the average context length from under 700 words in FinQA to 123k words in DocFinQA. |
Eliciting In-context Retrieval and Reasoning for Long-context Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks like LOFT often overestimate LCLM performance by providing overly simplified contexts. |
| Approach: | They propose to use retrieval-attention-probing to filter and de-noise long contexts during decoding and joint retrieval head training alongside the generation head to improve LCLM performance. |
| Outcome: | The proposed approach outperforms RAG and GPT-4-Turbo on most tasks despite being a much smaller model. |
Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QA (2024.emnlp-main)
Copied to clipboard
Minzheng Wang, Longze Chen, Fu Cheng, Shengyi Liao, Xinghua Zhang, Bingli Wu, Haiyang Yu, Nan Xu, Lei Zhang, Run Luo, Yunshui Li, Min Yang, Fei Huang, Yongbin Li
| Challenge: | Existing benchmarks for evaluating long-context language models employ irrelevant noise texts to artificially extend the length of test cases, diverging from the real-world scenarios of long-constituency applications. |
| Approach: | They propose a long-context benchmark, Loong, aligning with realistic scenarios through extended multi-document question answering (QA) . |
| Outcome: | The proposed model can scale up the context window of large language models to perform in-depth analysis of multiple long documents. |
MeetingQA: Extractive Question-Answering on Meeting Transcripts (2023.acl-long)
Copied to clipboard
| Challenge: | Meeting transcripts are a promising domain for natural language tasks . lack of annotated data impedes research on other important tasks in this domain . |
| Approach: | They propose an extractive QA dataset comprising questions asked by meeting participants and corresponding responses. |
| Outcome: | The proposed dataset extracts questions asked by meeting participants and corresponding responses from transcripts. |
Efficient Context Selection for Long-Context QA: No Tuning, No Iteration, Just Adaptive‐k (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing adaptive methods struggle with aggregation QA where optimal external context is unknown and variable. |
| Approach: | They propose a single-pass method that selects a query-specific number of passages . Adaptivek retrieval matches or outperforms fixedk baselines while using 10x fewer tokens compared to full-context input . |
| Outcome: | Adaptivek retrieval matches or outperforms fixedk baselines on factoid and aggregation QA benchmarks . it uses 10x fewer tokens than full-context input and still retrieves 70% of relevant passages compared to previous methods . |
Query-Focused Retrieval Heads Improve Long-Context Reasoning and Re-ranking (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent work has identified retrieval heads as a subset of attention heads responsible for retrieving salient information in long-context language models. |
| Approach: | They introduce a retrieval head that uses attention scores to enhance retrieval from long context . they use QRRetriever to select the most relevant parts with the highest retrieval scores . |
| Outcome: | The proposed retrieval heads outperform other retrieval-based retrieval retrievers on BEIR benchmarks. |
Benchmarking Long-Context Language Models on Long Code Understanding (2025.acl-long)
Copied to clipboard
Jia Li, Xuyuan Guo, Lei Li, Kechi Zhang, Ge Li, Jia Li, Zhengwei Tao, Fang Liu, Chongyang Tao, Yuqi Zhu, Zhi Jin
| Challenge: | Currently, long-context language models are limited by the lack of a rigorous evaluation framework for long code understanding. |
| Approach: | They propose to use a long code understanding benchmark LongCodeU to evaluate LCLMs' long code comprehension ability for practical applications. |
| Outcome: | The proposed benchmarks show that current LCLMs are limited in their long code understanding ability, particularly when the long code length is greater than 32K, falling far short of their claimed 128K to 1M context windows. |
Stronger Baselines for Retrieval-Augmented Generation with Long-Context Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing long-context language models (LMs) can handle tens of thousands of tokens in a single context window. |
| Approach: | They compare two recent multi-stage pipelines, ReadAgent and RAPTOR, against three baselines. |
| Outcome: | The proposed pipelines outperform more complex methods on multiple long-context QA benchmarks. |